Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-27744][SQL] preserve spark properties on async subquery tasks #24625

Closed
wants to merge 1 commit into from

Conversation

onursatici
Copy link
Contributor

What changes were proposed in this pull request?

Add a new future helper method that copies over spark properties before submitting to a new thread, and sets the original properties back on exit. SubqueryExec now uses this method to create futures. Any task that is submitted to its thread pool will now use the spark properties of the submitting thread, instead of the spark properties that were on the thread pool worker thread

How was this patch tested?

Added a test, which fails with current master but passes with this patch

@onursatici
Copy link
Contributor Author

@gatorsmile curious to hear your thoughts about this

@onursatici
Copy link
Contributor Author

friendly ping @gatorsmile

@onursatici
Copy link
Contributor Author

friendly ping @gatorsmile

@gatorsmile
Copy link
Member

ok to test

@SparkQA
Copy link

SparkQA commented Jun 5, 2019

Test build #106204 has finished for PR 24625 at commit 4b765bf.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@onursatici
Copy link
Contributor Author

@gatorsmile do you have any comments? should I tag anyone else for review? thanks!

@onursatici
Copy link
Contributor Author

friendly ping on above @gatorsmile

bulldozer-bot bot pushed a commit to palantir/spark that referenced this pull request Jun 20, 2019
## Upstream SPARK-XXXXX ticket and PR link (if not applicable, explain)
https://issues.apache.org/jira/browse/SPARK-27744
apache#24625
## What changes were proposed in this pull request?
preserve spark properties on subquery exec tasks

## How was this patch tested?
unit tests
@onursatici
Copy link
Contributor Author

friendly ping on above @gatorsmile

@AmplabJenkins
Copy link

Can one of the admins verify this patch?

@github-actions
Copy link

We're closing this PR because it hasn't been updated in a while.
This isn't a judgement on the merit of the PR in any way. It's just
a way of keeping the PR queue manageable.

If you'd like to revive this PR, please reopen it!

@github-actions github-actions bot added the Stale label Dec 29, 2019
@github-actions github-actions bot closed this Dec 30, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants